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AM APPROXIMATELY REPRODUCING SCORING SCHEME THAT 
ALIGNS RANDOM RESPONSE AND OMISSION 

ABSTRACT 

One formulation of confidence scoring requires the examxnee to xndicate 
as a number his personal probability of the correctness of each alternatxve 
in a multiple-choice test. For this formulation, a linear transformatxon of 
the logarithm of the correct response is maximized if the examxnee reports 
accurately his personal probability. To equate omits scores with choice 
scores, the transformation can be chosen so that the score is zero if the 
examinee indicates complete uncertainty. If this is done, the scorxng functxon 
depends on the number of alternatives. One could also align uncertaxnty and 
response omission by granting credit for omitting items, though it xs felt tiu.s 
might be hard to explain to examinees. 



O 

ERIC 



AN APPROXIMATELY REPRODUCING SCORING SCHEME THAT 
aligns random RESPONSE AND OMISSION^ 

The related problems of guessing and partial knowledge have stimulated 
quite a lot of consideration by test-oriented persons who are dissatisfied 
with the limited amount of information conveyed by the responses to multxple- 
choice items. One way to increase this information without increasing the 
amount of substantive interpretation required is to allow the examinee to indicate 
for each alternative the amount of uncertainty, or probability, of correct- 
ness of each alternative. In so doing, one may make the testing process 
more palatable in that the examinee is allowed to communicate his unsureness 
and hence reduce the presumed feelin of risk and anxxety assocxated wxth 
marking the "best" answer— he may have very mixed feelings about the "bestness" 

of that answer. 

It should be mentioned that while there is much interest in improving 
testing procedures^ and confidence testing is s,trongly suggested by some 
(Shuford & Massengill, 1967), confidence testing should ' ’ ' .aceu 

uncritically as an improvement. Some have reservatxons whxch stem from the 
fact that confidence testing requires the examxnee to decxde whether to 
a risk and how much risk to take when making each response, as wxll be seen. 

With the usual multiple-choice testing, this decxsxon about possxble rxsk 
may be less apparent to the examinee, and, hence, the personality factors 
operative in the two types of testing may not be the same. Swineford (1938, 

1941 ) has presented evidence of a relation between personality factors 

and risk taking in confidence testing quite apart from achievements involved. 
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Therefore, one should take care to ascertain that the changed operations of 
personality factors introduced through confidence testing do not defeat the 
purpose of measureinent • 

The present paper is not responsive to the problem of personality 
factors but to the treatment of omitted responses. That is, it remains 
usual to coordinate omissions scoring with the rest of the scoring procedures, 
and that is the function of this paper, at least for the conf xdence-testing 
format discussed below. This format is one in which the examinee indicates 
his certainty of the correctness of each alternative as a nonnegative number, 
and the certainties recorded must sum to specified totpl, such as unxty in 
the case where they are described as being probabilities of correctness. 

De Finetti (1962) has raised the question as to whether when this is done, 
the examinees will give a response directly indicative of thexr personal 
probabilities of the correctness of the responses and has introduced some 
scoring functions that are maximized when the responses equal those personal 
probabilities (De Finetti, 1965)-the not^'u beir r. lonnl man will re- 

spond honestly when such behavior optimizes his expected score. Shuford, Albert, 
and Massengill (1966) introduced a formalization of this norxxn , called the 
reproducing scoring property, and have pointed out that wherr r e scores only 
the correct response, the scoring function which is reproduc±og is unique 
and is of the form 



S = A log B X , 



( 1 ) 
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where S is the item score and x is the response to the correct alternatxve. 
They have taken B as 10 and A as unity when the logarithm is to the base 
ten and fntroduced the arbitrary score of minus one when x is in an interval 



below one hundredth (so that the scoring function will be bounded) . Thus 

= 1 + log X, .01— X— 1; (2) 

in their formulation. 

These choices may be overly arbitrary, however, in that no provision 
is made for the situation where the examinee omits the item. For example, 
his score on an item about whose answer he hasn't the foggiest notion should 
be the smie whether he responds to it by telling that he knows nothing about 
it, or whether he omits it. He should also not expect to receive more credit 
for marking at random at the end of a test than the examinee who does not. 

To correct for omissions one might use formula (2) and assign a nonzero 
value to the omitted items. For example, in a four- choice test the value of 

= 1 + log . 25 

or about .4 is the score to be assigned to each omitted item. For 2- ^ 
and 5-choice items the scores assigned to omits would be about .7, .5, 
and .3 respectively. If these corrections for guessing are used, they may, 
however, still prove unsatisfactory iu that the examinee may have some diffi- 
culty understanding why points should be given for omits and might adapt 
some truly pathological strategy out of misunderstanding unless he thinks 
that omits will be physically ignored in the scoring process. 

When using traditional formula scoring, one sets up the; formula so that 
the average score under random guessing is zero, and it is suggested here 
that such could also be done in the confidence testing situation by appro- 
priate choice of A and B in the scoring function. This is done by setting 
B equal to the number of alternatives. Then when the examinee marks that 
his uncertainty is 1/k where k is the number of alternatives, as he would 
if he is indicating no information, the score would be the same as if he 



omitted it in that either way the score is zero 



Thus the formula 



S, = A (log k + log x) 



( 3 ) 



takes on a aero when uncertainty Is expressed and does so no matter which 
alternative the examinee marks. Table 1 Is provided with entries aligned 
with a zero assignment to omits, and the value used £or the constant A is 

A = 1/ (log k) 

which sets the upper bound of the score at unity. At the lower range of 
the table where x approaches zero, the value of the scoring function when 

X is .0 is used to keep the function bounded. 

The alignment provided by adjusting the score for omits as suggested 
either way does not allow one to distinguish the situation where a nonchance 
level of uncertainty Is assigned to some other alternative from one where 
all the responses are at the chance level. To handle this situation using 
only one response per Item, one might score only the highest certainty reward- 
ing the response differently when it Is right than when It Is wrong (Boldt, 
1971) . When this Is done a chance response would indicate complete certainty 

since the certainties must sum to one. 
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Footnotes 

^This research was supported by the Technical Training Division, 

Air Force Human Resources Laboratory, Lowry Air Force Base, Colorado. This 
report was issued by them as AFHRL-*TM^12 . 

^The development of formula (1) can be carried out as follows. Let 
be the score assigned if alternative h is correct and the examinee has 
indicated an amount of certainty equal to r^ . Then if is his subjectxve 

probability that alternative 'h. is correct, his expected score over all 

I 

( 

alternatives is 

h 

and it is desired to have E at a maximum when r^ = p^ subject to the 



constraint that 



tr. = 1 . 

h " 



Thus the objective function 



h n 



where A is the Lagrange multiplier imposing the condition that the r s 



sum to one^ is maximized when 
Pt 






^h dr. 



= A, 



’^h == Ph 



or 

A. 

■'■’h ■ '“h ■ 

Therefore, E is at a maximum when the scoring function, S , is 
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where x is the indicated certainty for the correct answer , and 
constant of integration. The proof is ancillary to the text of 
but is included as it is quite a bit sratpler than that given by 
et al.(1966). 
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Table 1 

Score Certainty (%) of Correctness 
of Alternative Keyed Correct 



Confidence 


Number of Alternatives 




2 


3 


4 


5 


0 


-56 


-32 


-23 


-19 


5 


-33 


-17 


-12 


- 9 


10 


-23 


-11 


- 7 


- 4 


15 


-17 


- 7 


- 4 


- 2 


20 


-13 


- 5 


- 2 


0 


25 


-10 


- j 


0 


1 


30 


- 7 


- 1 


1 


3 


1/3 


- 6 


0 


2 


3 


35 


- 5 


0 


2 


3 


40 


- 3 


2 


3 


4 


45 


- 2 


3 


4 


5 


50 


0 


4 


5 


6 


55 


1 


5 


6 


6 


60 


3 


5 


6 


7 


65 


4 


6 


7 


7 


2/3 


4 


6 


7 


8 


70 
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75 
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7 
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80 
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8 
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85 
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90 
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95 
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10 


10 


10 


100 


10 


,10 


10 


10 





